Have Geek Brain

What's the best way to manage multiple media servers and distribute files between them?

I have a file host file that writes over 2MB bandwidth, so I need to start adding secondary media servers to store the files. What would be the best way to manage the setup of multiple servers with a lot of files? Preferably only via php.

Currently I only have about 100Gb of files ... so I could get a 2nd server, mirror all the content in between, and then combine the traffic 50/50, 33/33/33, etc. But once the total number of files exceeds the capacity of one server, it won't work.

The idea I had was to have a list of media servers stored in the DB, with the amount of free space remaining on each server. After uploading the file, php will select the server where the file is actually uploaded and distribute all files evenly between the servers.

Was hoping to get even more information / inspiration.

Don't use any third party services like Amazon. Files range from a few bytes to a gigabyte.

thanks

+1


a source to share


5 answers


If you do as much data transfer as you say, it would seem that everything you do grows rather quickly.

You might want to contact your hosting provider and see if they offer any general storage solutions via iscsi, nas, or other means. Ideally, storage will not only start so big that it will store everything you have, it will dynamically grow beyond your needs. I know my hosting provider offers such a solution.

If they don't, you might consider hosting your servers somewhere, either offering such a service, or letting you install your own storage server (which could be built cheaply with complete components and software. such as Freenas or Openfiler).

Once you have a centralized storage platform, you can add web servers to your content and load them based on load while accessing the same central storage.



Not only is this the right way to do it, but it will give you a lot more redundancy and extensibility in the future if you continue to grow at the rate that you are currently growing.

Other solutions offered using a database repository of what is stored in it will work, but this not only adds an extra layer of complexity to the fold, but also an extra layer of processing between your visitors and the data they want.

What if you lost your hard drive, will you lose 1/3 or 1/2 of all your data?

Should the heavy IO of static content be on the same spindles as the rest of your operating system and application data?

+1


a source


You can try MogileFS . It is a distributed file system. Has a good PHP API. You can create categories and upload a file to that category. For each category, you can define how many servers should be allocated. You can use the API to get the url of this file in a random node.



+2


a source


It's best to have your files in some kind of storage that scales. Storing files locally should only be done for serious reasons (sensitive, private, etc.).

Your best bet is to move your content to the cloud. Mosso CloudFiles or Amazon S3 lets you store an almost infinite number of files. Then all your content is available through the API If you want, you can use MySQL to track metadata for easy retrieval and let the service handle the actual file storage.

0


a source


I think your own idea is not the worst. get a bunch of servers and for each file store that he (a) server. if new files are uploaded, use most of the free space first *. each server handles its own delivery (instead of routing through the main server).

pros:

  • use multiple servers for one file. eg for cutekitten.jpg: filepath="server1\cutekitten.jpg;server2\cutekitten.jpg"

    and then select the server depending on the server load (either randomly, or alternating, ...)

  • If you're careful, you should be able to automatically move files based on your current load. so if your cute kitty image gets renamed / slashdotted, move it to the lightest server and update the post.
    you can do it with cron-job. just log downloads in the last xx minutes. try some formal ones like (downloads-per-minutefilesize (server download product)) for weighting. select tresholds to increase / decrease the number of servers these files are propagated to.

  • if you add a new server it is relativistic and painless (just add the address to the server pool)

minuses:

  • home solutions are always dangerous

  • your load balancing algorithm must be well tested, otherwise bad things can happen (everything is mirrored everywhere)

  • constantly moving files for balancing adds additional load to the server

*

or use a mixed weighting algorithm: free space, server load, file popularity

disclaimer: never been in a situation myself, just guessing.

0


a source


Consider HDFS, which is part of Apache Hadoop. This will integrate with PHP, but you will be creating a second application. It will also resolve all your server-side balancing issues and handle things when your file space usage exceeds one server. It's not pure PHP, though, but I don't think what you meant when you said "pure" anyway.

See http://hadoop.apache.org/core/docs/current/hdfs_design.html for an idea. They cover the whole idea of ​​how it handles large files, many files, replication, etc.

0


a source






More articles:


All Articles