theluyuan/RWKV-Runner
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

improve memory usage of state cache

Browse Source
This commit is contained in:
josc146
2023-10-28 23:04:49 +08:00
parent 77df56cddc
commit ff7306349a
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
backend-python/routes/state_cache.py
View File

@@ -9,7 +9,7 @@ router = APIRouter()
trie = None trie = None
dtrie: Dict = {} dtrie: Dict = {}
max_trie_len = 3000 max_trie_len = 300
loop_start_id = 1 # to prevent preloaded prompts from being deleted loop_start_id = 1 # to prevent preloaded prompts from being deleted
loop_del_trie_id = loop_start_id loop_del_trie_id = loop_start_id