mirrors/tabby
1
0
Fork 0
mirror of https://github.com/TabbyML/tabby synced 2024-11-22 00:08:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: handle 429 Too Many Requests status code in crawl_url function (#2539)

Browse Source
This commit is contained in:
Meng Zhang 2024-06-28 15:26:17 +08:00 committed by GitHub
parent c3e9f624b2
commit 894a4c6c7c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
crates/tabby-scheduler/src/crawl/mod.rs
View File

@ -54,6 +54,10 @@ async fn crawl_url(start_url: &str) -> anyhow::Result<impl Stream<Item = KatanaR
}
};
if data.response.status_code == Some(429) {
logkit::warn!("429 Too Many Requests, consider adjust your rate limit settings...");
}
// Skip if the status code is not 200
if data.response.status_code != Some(200) {
continue;