Training resource list
-
Real dialogue between 200k patients and doctors, source HealthCareMagic.com HealthCareMagic-200k . -
26k Real dialogue between patients and doctors, source icliniq.com icliniq-26k . -
5K generated dialogue between patients and doctors, from ChatGPT GenMedGPT-5k and disease database .
-
University of Texas Southwestern Medical Center, Dallas, USA -
University of Illinois at Champaign, Urbana, USA -
Ohio State University, Columbus, USA -
Hangzhou University of Electronic Science and Technology, Hangzhou, China
Operational Guidelines
pip install -r requirements.txt
How to fine tune
torchrun --nproc_per_node=4 --master_port=<your_random_port> train.py \ --model_name_or_path <your_path_to_hf_converted_llama_ckpt_and_tokenizer> \ --data_path ./ HealthCareMagic-200k.json \ --bf16 True \ --output_dir pretrained \ --num_train_epochs 3 \ --per_device_train_batch_size 4 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 8 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 2000 \ --save_total_limit 1 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --fsdp "full_shard auto_wrap" \ --fsdp_transformer_layer_cls_to_wrap 'LLaMADecoderLayer' \ --tf32 True
