Commit Graph

780 Commits

Author SHA1 Message Date
Wei Zhang
97210ece9a
feat(context): add support for Pull Request in GitHub Context (#3429)
* WIP: add pr answer mock

* WIP: rename diff to patch

* WIP: index pulls and answer from them

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* chore: use diff instread of patch

* chore: drop PR placeholder in answer engine

* [autofix.ci] apply automated fixes

* chore: use consistent pull and pullDoc

* chore(graphQL): update schema for pr context

* chore: minor update for comments

* chore: skip diff larger than 10MB

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-11-19 10:57:32 +08:00
Meng Zhang
4787ef78ae
feat(languages): add support for txt language with appropriate extens… (#3426) 2024-11-15 18:03:36 +08:00
Meng Zhang
a52c4e6fe7
refactor(index): migrate corpus::WEB to corpus::STRUCTURED_DOC (#3352)
* add structured doc

* [autofix.ci] apply automated fixes

* chore: implement structured_doc::DocService

* refactor(index): refactored `web_crawler.rs` to use updated `StructuredDoc` and `StructuredDocFields` types.

run make fix

* switch doc search

* chore: adapt frontend

* delete doc related files

* run make fix

* add deprecation notes for corpus::WEB

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-11-13 20:13:51 -08:00
Jackson Chen
414cac3847
fix(codeSearch): normalize path for code search (#3393)
* chore: normalize file paths in code search query

* chore: adding debug

* chore: Normalize file paths in code search query

* chore: fix some chore change

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* to: adding tests to normalize path

* chore: Normalize file paths in code search query

* to: Normalize file paths in code search query

to: remove unuse import

* chore: adding comment slash

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-11-13 18:13:02 -08:00
Meng Zhang
ca61582111
feat(tabby-index): add filepath embedding tokens into code index (#3413)
* feat(tabby-index): add filepath embedding tokens into code chunk

* refactor(answer): optimize file content handling and hit merging

- Simplified file content reading logic by removing unnecessary checks.
- Limited the number of hits per file to a maximum of 2.

* refactor(code): limit code search hits to at most two per file and increase LoC threshold

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* refactor(code): simplify token handling and improve chunk body formatting

* [autofix.ci] apply automated fixes

* chore: test retain_at_most_two_hits_per_file (#3418)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Wei Zhang <kweizh@tabbyml.com>
2024-11-13 17:31:22 -08:00
Wei Zhang
a7d29b822c
fix(models-http-api): use completion and embedding endpoint for llama.cpp (#3409) 2024-11-12 07:56:10 -08:00
Meng Zhang
4a26fc1f4a Release 0.21.0-dev.0
aim-downloader@0.21.0-dev.0
hash-ids@0.21.0-dev.0
http-api-bindings@0.21.0-dev.0
llama-cpp-server@0.21.0-dev.0
ollama-api-bindings@0.21.0-dev.0
tabby@0.21.0-dev.0
tabby-common@0.21.0-dev.0
tabby-crawler@0.21.0-dev.0
tabby-db@0.21.0-dev.0
tabby-db-macros@0.21.0-dev.0
tabby-download@0.21.0-dev.0
tabby-git@0.21.0-dev.0
tabby-index@0.21.0-dev.0
tabby-index-cli@0.21.0-dev.0
tabby-inference@0.21.0-dev.0
tabby-schema@0.21.0-dev.0
tabby-webserver@0.21.0-dev.0

Generated by cargo-workspaces
2024-11-09 21:46:51 -08:00
antimonyGu
28a5328d9f
feat(ui): allow selecting model in answer engine (#3304)
* llm select FE sketch

* fetch model array & select model drop down

* finish model select function & to be polished

* fix selectedModel init value and style

* fix lint

* rename DropdownMenuItems

* refine feature toggle

* chore(answer): set chat model's name by selection

* refine model select dropdown position

* [autofix.ci] apply automated fixes

* properly fill model name

* rename selectedModelName to modelName

* uplift modelName state

* clean log

* format code

* fix lint

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* [autofix.ci] apply automated fixes (attempt 3/3)

* feat(ui): show tool bar in  search page

* fix: fix ui test

* refactor(chart-ui): persist modelName and show modelName selection in followup

* handle selectedModel not in models api

* fix: throw warning when request model is not in supported_models

* fix: fix modelInfo?.chat not supported case and check request.mode is supported in BE

* fix: using warn! to print warning

* fix: fix ui lint

* update: ajust style and check if model is valid

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Meng Zhang <meng@tabbyml.com>
Co-authored-by: liangfung <1098486429@qq.com>
2024-11-05 08:16:21 -08:00
Meng Zhang
27f45f9213
chore(llama-cpp-server): add timeout and retry delay for llama-server health check (#3369) 2024-11-05 09:51:48 +08:00
Wei Zhang
dd533c8530
fix(download): should check local models validate download urls (#3363)
* fix(download): should check local models validate download urls

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-11-03 14:19:18 -08:00
antimonyGu
8ffec60ee5
chore(config): validate supported models (#3293)
* validate supported models

* fix comments

* clean code

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* Unit test for Config::validate_model_config

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-11-01 12:43:27 -07:00
Meng Zhang
8e67917063 docs(indexer): add documentation for build_attributes and build_chunk_attributes methods 2024-10-31 15:24:46 -07:00
Meng Zhang
9c8e6fd26e refactor(api): remove unused serde and utoipa attributes for CodeSearch / DocSearch 2024-10-31 15:24:46 -07:00
Wei Zhang
84787e0099
chore(llama.cpp): bump version to b3995 (#3347) 2024-10-31 09:06:44 -07:00
Meng Zhang
e8cb4ce5fb Release 0.20.0-dev.0
aim-downloader@0.20.0-dev.0
hash-ids@0.20.0-dev.0
http-api-bindings@0.20.0-dev.0
llama-cpp-server@0.20.0-dev.0
ollama-api-bindings@0.20.0-dev.0
tabby@0.20.0-dev.0
tabby-common@0.20.0-dev.0
tabby-crawler@0.20.0-dev.0
tabby-db@0.20.0-dev.0
tabby-db-macros@0.20.0-dev.0
tabby-download@0.20.0-dev.0
tabby-git@0.20.0-dev.0
tabby-index@0.20.0-dev.0
tabby-index-cli@0.20.0-dev.0
tabby-inference@0.20.0-dev.0
tabby-schema@0.20.0-dev.0
tabby-webserver@0.20.0-dev.0

Generated by cargo-workspaces
2024-10-30 21:42:38 -07:00
Wei Zhang
86132a88d9
docs(inference): document completion layered traits (#3340)
* chore(doc): clear completion layered architecture

* chore: fix grammer

Co-authored-by: Meng Zhang <meng@tabbyml.com>

---------

Co-authored-by: Meng Zhang <meng@tabbyml.com>
2024-10-30 18:06:51 +00:00
Meng Zhang
5098234ec2
refactor: run make fix for rust code (#3339) 2024-10-30 17:49:36 +00:00
Wei Zhang
910367fecb
fix(completion): should build fim prompt for openai aliases (#3341)
* fix: should build fim completion prompt for openai aliases

* chore: update fim list name

Co-authored-by: Meng Zhang <meng@tabbyml.com>

---------

Co-authored-by: Meng Zhang <meng@tabbyml.com>
2024-10-31 00:54:32 +08:00
Wei Zhang
d4c5d154ee
refactor(openai): support fim and no_fim for openai completion (#3338)
* refactor(openai): mark suffix support legacy, use prompt as default

* [autofix.ci] apply automated fixes

* chore: specify more completion vendors

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-29 19:53:29 -07:00
Wei Zhang
a3a6bea000
feat(graphQL): add model check for chat, completion, embedding (#3311)
* feat(graphQL): add model health check for chat

* feat(graphql): add completion test connection

* feat(model): test embedding model connection

* chore: update graphql schema to return latency

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* chore: ignore use schema result lint

* choer(graphql): use response and error type

* chore: use CompletionStream for completion health check

* chore: specify chat check message to response quick

* chore: use TestModelConnectionError

* [autofix.ci] apply automated fixes

* chore: fix review from meng

* [autofix.ci] apply automated fixes

* update

* [autofix.ci] apply automated fixes

* update

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Meng Zhang <meng@tabbyml.com>
2024-10-29 22:05:14 +00:00
Meng Zhang
fcb367e04a
fix(config.answer): properly implement Default for AnswerConfig to us… (#3336)
* fix(config.answer): properly implement Default for AnswerConfig to use customized default value

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-28 16:33:12 -07:00
Wei Zhang
0e75cb2966
chore(completion): replace CRLF with LF for code completion LLM requests (#3303)
* feat: replease crlf to lf in completions

* [autofix.ci] apply automated fixes

* chore: crlf logic should under service

* [autofix.ci] apply automated fixes

* chore: replace CRLF for prompt instead of segments

* [autofix.ci] apply automated fixes

* chore: use functions and ut for replacing crlf

* chore: rename generated to generated_text

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-27 18:27:42 -07:00
Meng Zhang
df8ce08051
chore(config): modify default function for code search params in answer engine (#3321) 2024-10-24 21:49:57 -07:00
Wei Zhang
681dc9d17b
fix(download): should not error when the model to be removed not existed (#3296)
* fix: should not be error if file not found when newly download model

* chore: only check sha256 when model exists
2024-10-22 10:05:02 +08:00
Wei Zhang
ff3410d7fd
chore(download): properly implement partitioned model download (#3294)
* bug: fix ignored local model

* feat: should redownload if part of models not found

* test: fix sha256 check returned error

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-21 16:19:54 +08:00
Wei Zhang
779f785827
feat(download): allow fetching model files with multiple partitions (#3258)
* finish main logic

* add ut

* [autofix.ci] apply automated fixes

* feat: use indexed model name

* chore: apply review from meng

* chore: revert unnecessary downloader change

* chore: fix ut

* chore: donwload one file each time

* [autofix.ci] apply automated fixes

* chore: fix ut

* chore: fix review from meng

* chore: fix ci

* chore: revert multibar

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* chore: filter download address tests should be serial

* chore: use the workspace dep for serial_test

---------

Co-authored-by: leili <lilei@deeproute.ai>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-21 13:34:03 +08:00
Meng Zhang
64ccce57f4
feat(config): add additional language configuration support (#3253)
* feat(config): add additional language configuration support

* [autofix.ci] apply automated fixes

* add feature changie

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-10 10:56:18 +08:00
Meng Zhang
f57068c5c3
feat(config): add system_prompt to AnswerConfig and update related tests (#3254)
* feat(config): add system_prompt to AnswerConfig and update related tests

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-09 12:26:23 +08:00
Meng Zhang
494cd31366 Release 0.19.0-dev.0
aim-downloader@0.19.0-dev.0
hash-ids@0.19.0-dev.0
http-api-bindings@0.19.0-dev.0
llama-cpp-server@0.19.0-dev.0
ollama-api-bindings@0.19.0-dev.0
tabby@0.19.0-dev.0
tabby-common@0.19.0-dev.0
tabby-crawler@0.19.0-dev.0
tabby-db@0.19.0-dev.0
tabby-db-macros@0.19.0-dev.0
tabby-download@0.19.0-dev.0
tabby-git@0.19.0-dev.0
tabby-index@0.19.0-dev.0
tabby-index-cli@0.19.0-dev.0
tabby-inference@0.19.0-dev.0
tabby-schema@0.19.0-dev.0
tabby-webserver@0.19.0-dev.0

Generated by cargo-workspaces
2024-10-08 16:43:25 +08:00
Jackson Chen
42e1cedc11
feat(core): adding opened recent file to segments struct (#3237)
* feat(backend): adding open recent file segments

chore: add relevant snippets to client-open-api

* feat: update snippet count check in extract_snippets_from_segments
2024-10-05 14:34:03 +08:00
Wei Zhang
780b9eb654
feat(core): support configuring stop words in model config (#3209)
* feat: add stop words to model config

* chore: use the name stop_words_from_config

* [autofix.ci] apply automated fixes

* chore: fix snap test due to source file change

* chore: use empty text in addtional stop words

* chore: remove dup with_stop_words

* chore: fix unit test for code splitter

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-10-05 12:41:50 +08:00
antimonyGu
906e2c2dfb
feat(core, webserver): add /v1/models api (#3164)
* draft: feat(core, webserver): add /v1/models api

* seperate models file

* update model api

* embedding model pattern match

* polish

* fix comments

* rename return structure

* [autofix.ci] apply automated fixes

* fix comments except pass state

* [autofix.ci] apply automated fixes

* pass config to model api through state

* [autofix.ci] apply automated fixes

* draft: to be fix shared reference

* fix config pass through state

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-30 11:20:09 +08:00
eryue0220
abdf72a251
feat(core): support customize presence_penalty for answer engine in config.toml (#3196)
* feat: update

* fix: cr
2024-09-26 17:02:31 +08:00
Meng Zhang
385d7ae093
refactor(core): remove CodeRepositoryAccess, add AllowedCodeRepositor… (#3185)
* refactor(core): remove CodeRepositoryAccess, add AllowedCodeRepository as axum Extension

* update

* Update ee/tabby-webserver/src/service/user_group.rs

* refactor(webserver): remove more useless fields in Webserver

* cleanup
2024-09-23 12:27:16 +00:00
Meng Zhang
531062a153
fix(core): fix ggml path loading for windows. fix user_agent field (m… (#3152)
* fix(core): fix ggml path loading for windows. fix user_agent field (mark as optional)

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-16 19:09:21 +08:00
Meng Zhang
1ef94ce6c8 fix(build): disable GGML_NATIVE explicitly (#3118) 2024-09-10 14:27:22 -07:00
Meng Zhang
d3519e5277
feat(webserver): implement user_groups CRUD apis (#3108)
* fix(db): enforce NOT NULL constraint on primary keys in access control tables

* feat(webserver): add user group management and membership update apis

* chore(webserver): implement UserGroupService.list

* make fix

* update graphql schema

* refactor(db): extract user_groups.rs

* [autofix.ci] apply automated fixes

* test(schema): add test cases for user group name regex

* test(user_groups): add unit test for UserGroupServiceImpl

* move access policy to service argument so it can be tested

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-10 10:57:13 -07:00
Meng Zhang
c9e61bc6e5 Release 0.18.0-dev.0
aim-downloader@0.18.0-dev.0
hash-ids@0.18.0-dev.0
http-api-bindings@0.18.0-dev.0
llama-cpp-server@0.18.0-dev.0
ollama-api-bindings@0.18.0-dev.0
tabby@0.18.0-dev.0
tabby-common@0.18.0-dev.0
tabby-crawler@0.18.0-dev.0
tabby-db@0.18.0-dev.0
tabby-db-macros@0.18.0-dev.0
tabby-download@0.18.0-dev.0
tabby-git@0.18.0-dev.0
tabby-index@0.18.0-dev.0
tabby-index-cli@0.18.0-dev.0
tabby-inference@0.18.0-dev.0
tabby-schema@0.18.0-dev.0
tabby-webserver@0.18.0-dev.0

Generated by cargo-workspaces
2024-09-05 19:45:32 -07:00
Meng Zhang
638ba37e83
refactor(hash-ids): fork hash-ids to return None instead of panic when ID is not valid (#3083)
* init commit

* update

* Update crates/hash-ids/Cargo.toml
2024-09-05 21:54:27 +00:00
Meng Zhang
c326d4c094
chore(indexing): propagate errors in repository cloning (#3067) 2024-09-04 17:52:57 +00:00
Meng Zhang
c8b1d2f49d refactor(crawl): remove useless extension filter as we apply it in -crawl-out-scope 2024-09-04 10:11:57 -07:00
Meng Zhang
2dbe2efbd2
fix(crawler): enhance crawling scope and exclusion patterns (#3065)
* chore(tabby-crawler): use breadth-first strategy for crawling

* fix(crawler): enhance crawling scope and exclusion patterns
2024-09-04 06:58:41 +00:00
Meng Zhang
d373f76dad
feat(crawler): add headless mode support based on environment variable (#3061)
* feat(crawler): add headless mode support based on environment variable

* refactor(tabby-crawler): enhance headless crawling options and rate limiting

* Update crates/tabby-crawler/src/lib.rs
2024-09-04 02:57:13 +00:00
Meng Zhang
66b5f8fc35
refactor(tabby-crawler): enhance crawling options and limits (#3058)
* refactor(tabby-crawler): enhance crawling options and limits

* Update crates/tabby-crawler/src/lib.rs

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-04 01:17:05 +00:00
Meng Zhang
75b5c58bd6
fix(web_documents): web crawler should take crawlerStart into conside… (#3048)
* fix(web_documents): web crawler should take crawlerStart into consideration for preset web documents

* fix id

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-02 21:34:15 -07:00
Meng Zhang
502b0525df
feat: add ocaml support (#2085)
* Add OCaml support.

* [autofix.ci] apply automated fixes

* Update programming-languages.md

* update

---------

Co-authored-by: O. Emmerson <oemmerson@gmx.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-09-01 23:06:52 +00:00
Meng Zhang
7ccbaee756
chore(webserver): update logging in code indexing and third-party integration (#3030)
* chore(webserver): update logging in code indexing and third-party integration

* update

* [autofix.ci] apply automated fixes

* Update ee/tabby-webserver/src/service/background_job/third_party_integration.rs

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-08-31 14:41:12 -07:00
Meng Zhang
ef1d0b2d87
chore(thread): adjust api interface to allow pass source_ids as query… (#3009)
* chore(thread): adjust api interface to allow pass source_ids as query constraint.

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-08-30 17:12:18 -07:00
Meng Zhang
b98b7f035c
feat(index): add chunk_size option to language configuration (#2990)
* feat(index): add chunk_size option to language configuration

* fix
2024-08-26 23:19:37 -07:00
Meng Zhang
03efbf626d
fix(index): fix broken code incremental indexing (#2988)
* fix(index): fix broken code incremental indexing

* [autofix.ci] apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-08-26 18:40:29 -07:00