View topic - Upcoming YafaRay v4

Upcoming YafaRay v4

General questions related with the YafaRay Project, 3D computer graphics and about this site.

Re: Upcoming YafaRay v4

Post Thu Mar 01, 2018 6:25 pm

Hello,

Thank you all for your excellent points supporting CPU.

The "only" problem I have now is that, to fully get advantage of Embree, I should use Ray packets and/or streams instead of single rays... and all Core integration is based on recursivity of single rays.

This means that to get better speeds I would also need to rewrite big parts of the integration code!

On one side that looks like a big work, but on the other side I would expect better speeds and also it would be a good learning process for me to better understand how integration works... but it will take time.

Otherwise I can stay for now with single rays, but reduced performance.. decisions, decisions... ;-)
User avatar
David Bluecame
 
Posts: 456
Joined: Mon Jan 21, 2013 12:42 pm
Location: Spain

Re: Upcoming YafaRay v4

Post Thu Mar 01, 2018 11:26 pm

Speed! We want speed! We demand it! :)

Seriously though, whichever you decide, we really appreciate the work you are doing, even when we're not saying it out loud :)
User avatar
stoneage
 
Posts: 1240
Joined: Sun Dec 23, 2007 1:06 pm

Re: Upcoming YafaRay v4

Post Fri Mar 02, 2018 7:30 pm

Is this something related to vcm integrator? I've heard only good things about it.

Regards
surena
 
Posts: 107
Joined: Sat Apr 26, 2014 6:37 am
Location: Spain

Re: Upcoming YafaRay v4

Post Mon Mar 05, 2018 2:52 am

David Bluecame wrote:Hello,

Thank you all for your excellent points supporting CPU.

The "only" problem I have now is that, to fully get advantage of Embree, I should use Ray packets and/or streams instead of single rays... and all Core integration is based on recursivity of single rays.

This means that to get better speeds I would also need to rewrite big parts of the integration code!

On one side that looks like a big work, but on the other side I would expect better speeds and also it would be a good learning process for me to better understand how integration works... but it will take time.

Otherwise I can stay for now with single rays, but reduced performance.. decisions, decisions... ;-)

Wow! It is good news. And when you release it? Any date?
Odilkhan Yakubov
 
Posts: 22
Joined: Thu Jun 19, 2014 11:44 pm

Re: Upcoming YafaRay v4

Post Fri Mar 23, 2018 9:57 am

We need news! :P
surena
 
Posts: 107
Joined: Sat Apr 26, 2014 6:37 am
Location: Spain

Re: Upcoming YafaRay v4

Post Sun Apr 29, 2018 9:01 pm

Hello,

Sorry that I've been absent from the forums for a while. Things at work have been extremely busy lately, with a lot of travelling involved :-/

Well, I'm sorry to dissappoint you but things are not going too well :-(

The new v4 architecture has several problems I'm now trying to solve:
* Embree is dissappointingly slow. I expected a significant speed increase respect to YafaRay Core but that's not always the case.
* RAM usage is higher than what I would desire.

Therefore I'm now working in several aspects:
* Reimplement YafaRay v3 intersection core in the new v4 architecture, so it's possible to choose between YafaRay intersector, Embree intersector and NanoRT (slow but just to give another choice) intersector. This way for some scenes, YafaRay intersector can still be used to get more speed in some cases.

* Totally overhauling the YafaRay v4 architecture, because re-integrating v3 was a nightmare and making the code too complex and difficult to maintain. Therefore I'm reorganizing all v4 files and folders to give the code a much more logical structure, to make development and maintenance easier.

* Getting rid of the "plugins" architecture! Even PBRT recommends to get away from it. In the past, many problems came from having the DLL "plugins" libraries separately from the main Core library, having to search for them, reloading them every frame... I cannot see any advantages on a plugins system anyway as there is no such thing as "3rd parties" generating plugins for yafaray and creating new plugins or modifying/extending the existing ones often require changes to Core. I hope this way, YafaRay can become better organized, more stable and much more streamlined (a single DLL for everything except for interfaces such as Python, Qt, etc.)

This is taking a very long time, but think that I'm trying to set the foundation for the future YafaRay development. The current source code structure comes from very old times and was never redesigned, causing it to be kind of difficult to understand and maintain. I hope this effort can help for the medium and long term.
User avatar
David Bluecame
 
Posts: 456
Joined: Mon Jan 21, 2013 12:42 pm
Location: Spain

Re: Upcoming YafaRay v4

Post Tue May 01, 2018 9:46 am

Take your time David, we are not in a hurry as the current version works well and extensive rewritings usually take their toll. Looking forward to beta test it as soon as you have something ready for the world to see.
User avatar
Samo
 
Posts: 3089
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: Upcoming YafaRay v4

Post Thu May 03, 2018 7:39 pm

Thank you, Alvaro. I do appreciate your answer... it's a quite frustrating process but I'm convinced it needed to be done, otherwise the complexity of the source code structure would be too much when opening the door to have different object/intersection kernels such as YafaRay native, Embree and who knows what else in the future...

I hope the code file structure will make more sense now and hopefully it will be easier to integrate new core functionality in the future.

I will keep you informed of my progress. To be honest, I never expected to go this deep, but strangely enough the most time consuming process for me so far has been to decide the best possible code architecture to integrate Embree, etc, not the integration itself (which I had tested months ago, but using "spaguetti code", not good for a serious release/maintenance and not optimizing RAM usage, etc).

Now I have to "reintegrate" once again all the changes I had made in the past, but with a better structure now...
User avatar
David Bluecame
 
Posts: 456
Joined: Mon Jan 21, 2013 12:42 pm
Location: Spain

Re: Upcoming YafaRay v4

Post Wed May 16, 2018 10:29 am

Hi David

I think that Spectre and Meltdown vulnerabilities have had a big impact in thread intensive app like YafaRay. For instance and after installing spectre and meltdown patches, now YafaRay is pretty much able to crash Windows 7 and Linux under heavy loads, alone or with other intensive processes, which is something I have never seen before. It is very likely that Embree performance is affected too, since easier branch prediction is one of its main goals.

Also I wonder if the code overhead to group coherent data is something you have to implement or it is something Embree does on its own, and whether packet raytracing implementation has been succesfull.
User avatar
Samo
 
Posts: 3089
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: Upcoming YafaRay v4

Post Wed May 16, 2018 4:05 pm

Hi, Alvaro.

Thank you for the info, I did not know!

Have you seen similar system crashes due to other renderers like Cycles, etc.

About Embree, to use ray packets I need to change the integrators code :-/
User avatar
David Bluecame
 
Posts: 456
Joined: Mon Jan 21, 2013 12:42 pm
Location: Spain

Re: Upcoming YafaRay v4

Post Thu May 17, 2018 7:31 am

Hi David

Yes I get crashes as well with other thread intensive app like Agisoft Photoscan and I think it is all down to meltdown and spectre patches, both Linux and Win systems have become much more unstable under multithreaded heavy loads.
User avatar
Samo
 
Posts: 3089
Joined: Tue Dec 20, 2005 10:39 am
Location: Spain

Re: Upcoming YafaRay v4

Post Sat May 19, 2018 6:14 pm

Hello,

Just some follow-up information about YafaRay v4.

As I have to re-integrate YafaRay v3 intersector in v4, I need to make the fundamental YafaRay code manageable. I'm working quite hard lately on reorganizing, "cleaning" and removing unnecessary complications from the YafaRay code before I re-merge the Embree code and the rest of the improvements I've been working on over the last 12 months.

My objective is to simplify the code base as much as I can, while keeping the current v3 functionality (and even optimizing and reducing a bit render times with the YafaRay native intersector).

I'm also working on defining a better and more clearly defined "public API". In fact, the next YafaRay will be mostly a single library I will call "libyafaray4.dll" for Windows and "libyafaray4.so" for Linux, etc. In addition to that library, we will have the libraries for Qt and Python/Ruby bindings that will still be in separate files.

I managed to make all the new libyafaray4 dependencies optional. So, if all the options are disabled, we will be able to fully build and test libyafaray4 without the need for any dependency whatsoever.

As for the problem of "how to test libyafaray4" without even XML libraries, well I have made a interesting addition to YafaRay. In addition to exporting scenes in XML as until now, we will be able to export scenes from Blender to Python .py files and as C++ source .cpp files. Those exported scene files can be rendered against libyafaray4 directly. The python files will need the python dependency and binding libraries, but the C++ scene source files can now be rendered to a "barebones" libyafaray4 library without any dependency whatsoever.

This should allow barebones yafaray4 usage and testing. That should make things much easier for debugging etc, as well as focus on a better defined (and much smaller and focused) public API that also 3rd party software can use.

So, even if all these improvements are not very spectacular, they should be the foundation for a new generation yafaray, more compact but at the same time more modular and expandable.

I will keep you informed of my progress. Sorry for the extremely slow development, but this is really difficult and I have limited time to work on YafaRay. In any case I hope it's worth it when this gets released (unless there aren't any YafaRay users left by that time ;-)

Best regards!
User avatar
David Bluecame
 
Posts: 456
Joined: Mon Jan 21, 2013 12:42 pm
Location: Spain

Re: Upcoming YafaRay v4

Post Tue Jun 26, 2018 2:47 am

Very exciting stuff my man, thanks for all the hard work.
400ml
 
Posts: 1
Joined: Mon May 16, 2011 12:34 pm

Previous

Return to News & Discussion



Who is online

Users browsing this forum: No registered users and 6 guests

cron