Upcoming YafaRay v4

[David Bluecame]

Hello,

Thank you all for your excellent points supporting CPU.

The "only" problem I have now is that, to fully get advantage of Embree, I should use Ray packets and/or streams instead of single rays... and all Core integration is based on recursivity of single rays.

This means that to get better speeds I would also need to rewrite big parts of the integration code!

On one side that looks like a big work, but on the other side I would expect better speeds and also it would be a good learning process for me to better understand how integration works... but it will take time.

Otherwise I can stay for now with single rays, but reduced performance.. decisions, decisions...

[stoneage]

Speed! We want speed! We demand it!

Seriously though, whichever you decide, we really appreciate the work you are doing, even when we're not saying it out loud

[surena]

Is this something related to vcm integrator? I've heard only good things about it.

Regards

[Odilkhan Yakubov]

Hello,

Thank you all for your excellent points supporting CPU.

The "only" problem I have now is that, to fully get advantage of Embree, I should use Ray packets and/or streams instead of single rays... and all Core integration is based on recursivity of single rays.

This means that to get better speeds I would also need to rewrite big parts of the integration code!

On one side that looks like a big work, but on the other side I would expect better speeds and also it would be a good learning process for me to better understand how integration works... but it will take time.

Otherwise I can stay for now with single rays, but reduced performance.. decisions, decisions...

Wow! It is good news. And when you release it? Any date?

[surena]

We need news!

[David Bluecame]

Hello,

Sorry that I've been absent from the forums for a while. Things at work have been extremely busy lately, with a lot of travelling involved :-/

Well, I'm sorry to dissappoint you but things are not going too well

The new v4 architecture has several problems I'm now trying to solve:
* Embree is dissappointingly slow. I expected a significant speed increase respect to YafaRay Core but that's not always the case.
* RAM usage is higher than what I would desire.

Therefore I'm now working in several aspects:
* Reimplement YafaRay v3 intersection core in the new v4 architecture, so it's possible to choose between YafaRay intersector, Embree intersector and NanoRT (slow but just to give another choice) intersector. This way for some scenes, YafaRay intersector can still be used to get more speed in some cases.

* Totally overhauling the YafaRay v4 architecture, because re-integrating v3 was a nightmare and making the code too complex and difficult to maintain. Therefore I'm reorganizing all v4 files and folders to give the code a much more logical structure, to make development and maintenance easier.

* Getting rid of the "plugins" architecture! Even PBRT recommends to get away from it. In the past, many problems came from having the DLL "plugins" libraries separately from the main Core library, having to search for them, reloading them every frame... I cannot see any advantages on a plugins system anyway as there is no such thing as "3rd parties" generating plugins for yafaray and creating new plugins or modifying/extending the existing ones often require changes to Core. I hope this way, YafaRay can become better organized, more stable and much more streamlined (a single DLL for everything except for interfaces such as Python, Qt, etc.)

This is taking a very long time, but think that I'm trying to set the foundation for the future YafaRay development. The current source code structure comes from very old times and was never redesigned, causing it to be kind of difficult to understand and maintain. I hope this effort can help for the medium and long term.

[Samo]

Take your time David, we are not in a hurry as the current version works well and extensive rewritings usually take their toll. Looking forward to beta test it as soon as you have something ready for the world to see.

[David Bluecame]

Thank you, Alvaro. I do appreciate your answer... it's a quite frustrating process but I'm convinced it needed to be done, otherwise the complexity of the source code structure would be too much when opening the door to have different object/intersection kernels such as YafaRay native, Embree and who knows what else in the future...

I hope the code file structure will make more sense now and hopefully it will be easier to integrate new core functionality in the future.

I will keep you informed of my progress. To be honest, I never expected to go this deep, but strangely enough the most time consuming process for me so far has been to decide the best possible code architecture to integrate Embree, etc, not the integration itself (which I had tested months ago, but using "spaguetti code", not good for a serious release/maintenance and not optimizing RAM usage, etc).

Now I have to "reintegrate" once again all the changes I had made in the past, but with a better structure now...

[Samo]

Hi David

I think that Spectre and Meltdown vulnerabilities have had a big impact in thread intensive app like YafaRay. For instance and after installing spectre and meltdown patches, now YafaRay is pretty much able to crash Windows 7 and Linux under heavy loads, alone or with other intensive processes, which is something I have never seen before. It is very likely that Embree performance is affected too, since easier branch prediction is one of its main goals.

Also I wonder if the code overhead to group coherent data is something you have to implement or it is something Embree does on its own, and whether packet raytracing implementation has been succesfull.

[David Bluecame]

Hi, Alvaro.

Thank you for the info, I did not know!

Have you seen similar system crashes due to other renderers like Cycles, etc.

About Embree, to use ray packets I need to change the integrators code :-/

[Samo]

Hi David

Yes I get crashes as well with other thread intensive app like Agisoft Photoscan and I think it is all down to meltdown and spectre patches, both Linux and Win systems have become much more unstable under multithreaded heavy loads.

[David Bluecame]

Hello,

Just some follow-up information about YafaRay v4.

As I have to re-integrate YafaRay v3 intersector in v4, I need to make the fundamental YafaRay code manageable. I'm working quite hard lately on reorganizing, "cleaning" and removing unnecessary complications from the YafaRay code before I re-merge the Embree code and the rest of the improvements I've been working on over the last 12 months.

My objective is to simplify the code base as much as I can, while keeping the current v3 functionality (and even optimizing and reducing a bit render times with the YafaRay native intersector).

I'm also working on defining a better and more clearly defined "public API". In fact, the next YafaRay will be mostly a single library I will call "libyafaray4.dll" for Windows and "libyafaray4.so" for Linux, etc. In addition to that library, we will have the libraries for Qt and Python/Ruby bindings that will still be in separate files.

I managed to make all the new libyafaray4 dependencies optional. So, if all the options are disabled, we will be able to fully build and test libyafaray4 without the need for any dependency whatsoever.

As for the problem of "how to test libyafaray4" without even XML libraries, well I have made a interesting addition to YafaRay. In addition to exporting scenes in XML as until now, we will be able to export scenes from Blender to Python .py files and as C++ source .cpp files. Those exported scene files can be rendered against libyafaray4 directly. The python files will need the python dependency and binding libraries, but the C++ scene source files can now be rendered to a "barebones" libyafaray4 library without any dependency whatsoever.

This should allow barebones yafaray4 usage and testing. That should make things much easier for debugging etc, as well as focus on a better defined (and much smaller and focused) public API that also 3rd party software can use.

So, even if all these improvements are not very spectacular, they should be the foundation for a new generation yafaray, more compact but at the same time more modular and expandable.

I will keep you informed of my progress. Sorry for the extremely slow development, but this is really difficult and I have limited time to work on YafaRay. In any case I hope it's worth it when this gets released (unless there aren't any YafaRay users left by that time

Best regards!

[400ml]

Very exciting stuff my man, thanks for all the hard work.

[David Bluecame]

Hello,

Sorry for the long absence and silence. I've been very busy with personal/work matters over the last months.

However, I'm still working on the upcoming v4. As the initial Embree changes/tests were... underwhelming... I had to revert back to the original YafaRay render engine and restart the process from it.

I've been working on changing the entire yafaray code structure to make more modular, streamlined and easy to maintain. I've also made a few changes to optimize the Render Passes code to make rendering a bit faster with the YafaRay render engine.

This process has been frustrating but I think it's worth it. I will have to work now on re-integrating the Embree code to coexist with YafaRay engine. As I'm getting YafaRay engine back, I will not use NanoRT this time, so it will be just "YafaRay or Embree" to be chosen by the user.

I'm making quite some interesting changes to the yafaray code and I have some surprises for you I'm sure you will like. However it's still too soon to tell you about them, I need to make them work properly first.

Best regards!

[Samo]

Thanks for the news David.