View topic - YafaRay Distributed Rendering (beta)

YafaRay Distributed Rendering (beta)

General questions related with the YafaRay Project, 3D computer graphics and about this site.

YafaRay Distributed Rendering (beta)

Post Tue May 05, 2009 12:23 pm

Hi all,

I created a distributed renderer to render a YafaRay scene with multiple computers. (Sorry, the renderer only works properly with windows for now. If I have time I'll port it to mono.)
It uses the GPU project( http://gpu.sourceforge.net ).
Here are some instructions on how to participate in rendering or to render your own scene:

How to participate:
  1. To do anything, you will need to download and install GPU. Download version 0.953 and install it.
  2. Don't start GPU yet, if you did, quit it now.
  3. Download http://tiltedentertainment.com/yafaray-gpu.1.0.1.exe and install it to the directory where you just installed GPU (Example: C:\Program Files\GPU\)
  4. When the installation has finished, start gpu
  5. Click on the text balloon icon to go to the chat screen.
  6. Take some time to set your name and country. There is a YafaRay Team in the teams list if you want to combine statistics.
  7. Now press start in the top left corner. You will start to see people appearing in the list on the right.
  8. Congratulations, you are now part of the distributed renderer. If you want to know how to render your own scenes, read on.
  9. To see what jobs are running on your system, press the Processor icon on the top and select the Jobs tab.

How to render your own scene:
  1. In order to render your own scenes you should first participate in the distributed renderer. If you haven't yet done so, follow the steps above.
  2. Go to the Advanced Plugins and Frontends tab and double click on YafaRay Renderer.
  3. You are now in the YafaRay Frontend. Here you can see the progress of rendered scenes, render scenes and add your own scenes for render.
  4. Go to the Add tab and add your scenes xml file.
  5. If you use textures, add them as well.
  6. Enter a Project Name.
  7. Enter your email (This is to make sure nobody deletes your scene (Still to be implemented) )
  8. To select the amount of pieces, think about how long the full image would take to render on an average computer.
  9. Each piece will only render for 5 minutes, if it isn't finished by then it is discarded and everything was for nothing.
  10. Select an amount of pieces so Average Render Time / Pieces < 5 minutes
  11. Press Create.
  12. If you see Upload succesfull, it worked.
  13. Go to the Render Tab and press Refresh.
  14. Select your project from the Project list.
  15. You should always try a local render first so you can watch for any error messages.
  16. Select Local render and press Single Render.
  17. If you go to the Jobs tab in the GPU window, you will see your render running.
  18. If the rendering works, you can start rendering it globally.
  19. Select Global render and press Start rendering.
  20. You can see the progress in the Display tab of the YafaRay Frontend.
  21. Select your project and you will see it build up slowly

Movies don't work yet, but will in the future.

If we all start rendering our own scenes at the same time everybody will have a much slower rendering. If you want to render images with GPU, please leave it on if your computer has nothing else to do. Rendering happens at idle priority so you won't notice it.

The renderer works best if you use sizes that are a multiple of 32 (1280, 1024, 800, 640, 480, 320 etc)

Bugs are constantly fixed, so if you want to have the most stable version, update often. (Redownload the installer and run it again)

Good luck rendering!

Thexa4

Update: Textures seem to work.

P.S. if rendering doesn't work, try to restart gpu.
Last edited by thexa4 on Thu May 07, 2009 4:12 pm, edited 7 times in total.
thexa4
 
Posts: 2
Joined: Tue May 05, 2009 11:32 am

Re: YafaRay Distributed Rendering (beta)

Post Tue May 05, 2009 1:30 pm

this sounds like a great idea!
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Re: YafaRay Distributed Rendering (beta)

Post Wed May 06, 2009 10:30 am

Ok guys. This thing works if people participate, al least a stable group of 10-20 available nodes is needed to make any difference. If not, it does not make much sense and this thing won't go anywhere. Also we need to throw some multi-core machines into the mix.

Besides, if you don't participate when you don't need it, don't expect people to assist you when you eventually need help from others with your renders.

In GPU, menu EDIT > Settings > Application, there are three options to enable GPU automatic startup, which are:

  • Run as service at operating system startup
  • Run minimized
  • Automatic connect when application launches

Please, enable these three options, it automatically launches GPU at system startup. It only uses unused cycles. Come on people, distributed rendering is quite compatible with current YafaRay algorimths!
Image

To developers:
If we really want to bring this to the masses, an integration effort should be done IMO. With this one, users have to use four different interfaces to use YafaRay: Blender UI for some parameters, python-coded settings UI, QT windows and now the YafaRay-GPU frontend.

It would be good if the distributed rendering could be launched from one of the existing interfaces. Also this integration should make possible frame rendering from Blender. I know it isn't that easy. But please think about it. Thank you.
User avatar
Samo
 
Posts: 3105
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: YafaRay Distributed Rendering (beta)

Post Wed May 06, 2009 10:49 am

Great! Looking forward to linux version :)
melon2008
 
Posts: 27
Joined: Wed Feb 13, 2008 10:14 am

Re: YafaRay Distributed Rendering (beta)

Post Wed May 06, 2009 4:44 pm

Hi!

Found it impossible to connect to the YafaRay-group with the GPU version 0.953 as downloaded from the GPU homepage.

I had to activate "Autoupdate to latest version" in the "Tools" menu in order to make it work...

Maybe this helps people with similar problems.

Greetings and keep up the great work!
IkariShinji
 
Posts: 176
Joined: Tue May 05, 2009 6:57 pm

Re: YafaRay Distributed Rendering (beta)

Post Wed May 06, 2009 4:55 pm

Samo

I only have a one computer and mainly render in OS X.

At work I have maybe 2, to 4 PCs I could install the software to run.
In case students do not close them.

Claas
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Re: YafaRay Distributed Rendering (beta)

Post Thu May 07, 2009 6:54 am

I believe the 'threads' issue is not fully adressed, I have a mono core no-HT pentium IV. By mistake, the xml was uploaded to the system with 'threads=3', which is what the original file has got.
gpu3.jpg
gpu3.jpg (13.1 KiB) Viewed 18614 times


As a result, when I have a job running in GPU, I have three yafaray-xml.exe running in my computer, all sharing the available cycles, which is 33% (when idle)

Obviously, my computer will have difficulties to finish any task within the 5-minutes limit if the available cycles all not well managed.

The second issue, when there is a job computing, and another job queuing, the queuing job does not queue in fact, and opens another three yafaray-xml.exe processes. So now every yafray-xml.exe is using a 15% of available cycles.

Besides there is another yafaray-xml.exe I don't know where the fuck it is coming from. Total, 7 yafaray-xml.exe running.
gpu2.jpg

gpu1.jpg

Also another issue is that sometimes there is only a job computing in the list or any job at all, but there are as many as 9 yafaray-xml.exe running in the task manager. It seems that the jobs aren't being killed even if they are not longer needed because they weren't deliver within the time limit. But that's bad if other jobs are assigned againg.
So what is the best practice about threads?
Also the queuing job should wait a bit and not stole cycles from the actual computing job.

My general impression is that, while the concept is sound and YafaRay is compatible with crop-based distributed rendering, the amount of feedback the cluster provides should be higher, users need more situational awareness. For instance whether a project is running, awaiting or definitely stopped, who is rendering what, what actions can be taken when a portion can't be apparently finished by the system, changing xml settings, files or amount of portions, without uploading a whole new project, deleting projects, manage priorities, restarting a project, etc, while keeping the 'freedom',anonimity and equalitarian spirit of the cluster.

Also the treads and jobs issue should be fixed.

Anyway is a good starting point an puts distributed rendering as a core issue for the YafaRay project.
User avatar
Samo
 
Posts: 3105
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: YafaRay Distributed Rendering (beta)

Post Thu May 07, 2009 4:10 pm

Mh I looked into it but I am not sure if it works right.

I can connect and upload my xml file.

When I click single render and local I see something pop up in GPU.
The same also when I click global render and start rendering.

However the entry stays only few seconds in the GPU job list and
than disappears.

If I go to the Yafaray GPU front end I can select the project but the
Frame preview is empty. Also when I click on reload nothing changes.
No rendered parts appear.

Also my GPU seems to stay in idle since I see Zzzzzz everywhere.
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Re: YafaRay Distributed Rendering (beta)

Post Thu May 07, 2009 5:07 pm

Try uploading a different scene, this one seems to be broken.
With what version of YafaRay did you create the xml file?
A version > 302 is recommended, I don't know if it works on older versions.
thexa4
 
Posts: 2
Joined: Tue May 05, 2009 11:32 am

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 1:25 am

I am running the latest Yafaray 305.


I still have some issues. I added GI to the scene to make it run longer.
And the job stays now. However I cannot see the result. There is
nothing inside the Display Yafaray Frontend. When I select the project
now a failed pops up to the right and the display preview area remains black.

I added my test file so you can see what I had setup.

I used a sun and skylight with direct light.
Attachments
yafrayRender.zip
(21.93 KiB) Downloaded 652 times
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 6:27 am

I believe there is something wrong with the latest version of GPU (0.954 update) and YafaRay, they do not seem to be compatible, and I don't get jobs or yafaray-xml.exe is launched when the 'start rendering' is pressed.

Whereas with default GPU 0.953 it seems to work better, inmediately after the 'Start Rendering' button is pressed, I get jobs and yafaray-xml.exe is launched.

Please can anyone else confirm?
User avatar
Samo
 
Posts: 3105
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 10:30 am

There is no linux version... so I can't help you
Bjo
 
Posts: 19
Joined: Fri Nov 21, 2008 5:17 pm

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 3:32 pm

is there a specific Yafaray GPU combiantion which should work?
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 5:19 pm

I believe the best approach is:

  • Pre render a portion of your image (shift+b), using the resolution the image is going to be divide into, when sent to the cluster to render. For instance, if an 800x600 image is going to be divided into 16 parts, pre render a portion of 200x150 (this info is given by Blender when entering the crop window)
  • Render that portion with final quality-wise settings.
  • This pre rendered portion should take a difficult area of the scene, with lot of geometry for instance.
  • This portion should take less than five minutes to be rendered.
If not, then:
  • Change scene sampling settings (light sampling, AO sampling, pathtracing sampling, FG sampling, AA sampling)
  • Change render final resolution.
  • Increase the number of parts the image is going to be divided into when sent to the cluster.

Untill you get less than five minutes to render the portion choosen. When you get less than five minutes, to render that part, send the project to the cluster.

BTW, GPU 0.954 works all right, but after a while if you don't get any jobs, even your own project's jobs, then restart GPU and the front end.
User avatar
Samo
 
Posts: 3105
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: YafaRay Distributed Rendering (beta)

Post Fri May 08, 2009 5:24 pm

I have to say I still do not grasp the requirements for the portions.

I can understand that each bucket should render below 5 minutes.

But the scenes I had were very simple and rendered in Yafaray+Blender
in less than 32 seconds.
...............................................
C l a a s E i c k e K u h n e n
Artist : Designer : Educator
User avatar
cekuhnen
 
Posts: 281
Joined: Mon Jan 10, 2005 10:00 pm

Next

Return to News & Discussion



Who is online

Users browsing this forum: No registered users and 1 guest

cron