File Transfering Design in Kanbox
1. What is file transfering?
Today, file transfering (such as uploading images, downloading images, sharing movies) is known well to everyone. With the mobile is more and more popular, file transfering on mobile phone is more and more important.
Django is a file transfering solution with features below:
- Safe: The file transfering can not be monitored by anyone else
- Fast: With the helper of CDN, user can download or upload file from the nearest server
- Smart: File is separated to pieces which will be uploaded one by one. Each chunk will be sec(validate exist) first, and then upload if not exist.
There are server roles in the Django Server:(from top to bottom)
- Django Client: The Django Client is an module(or a sdk) in the Kanbox app, it will upload or download file using Django Web API.
- CDN Server: There will be a Django module in each CDN server (which is part of tengine).
- Caching Files: There are many cache files in CDN server, client always download or upload file to CDN servers directly.
- Synchronize Cahces: The caches will be send to Django Server asynchronized with load shifting. And it can be send to Server synchornized when needed.
- Dual Write: Each file will not be deleted when is synchornized to server.
- Authorize: Requests from client will be authorize.
- Statistics: Each operation from client will be count and saved.
- Django Server: Django Server is the master server:
- Django Upload Cluster: Using MetaQ for offline task quene; Using Tair for saving meta data;Using TFS for backing up files.
- Django Download Cluster: Read and send file to client from TFS or other file system. There are MTS(media transfer system), DTS(doc), CTS(content) to transfer file to html.
- Django API Cluster: Supporting session and authorization.
3. Client Implementation
To support task arrangement, there are Task Manager in client which is responsible for dispatching task. And the Transfer Manager is responsible for transfer(download or upload) manage through http. All tasks will be save in database and thumbnail will be cache.
4. Sec Chunk & Upload
There are 5 step when uploading a file:
- Start Transaction
- Calculate MD5
- Sec Chunk
- Upload Chunk
- Commit Transaction
Each chunk’s size is 4MB. Several tasks can be run in parallel at the same time.