YafaRay architecture??

[David Bluecame]

Hello,

So far, all my yafaray_v4 development has been mainly tests changing the core object and intersection code, adapting it to Embree and doing some additional tests with new features.

However, my v4 test code is... messy. It's not the definitive version I want to upload to the repository and I want to make it right before doing so.

I have some questions for all of you about the decisions I have to face now regarding the internal YafaRay code architecture and the repercussions of these decisions for both users and developers.

The first big decision I have to make is this:
* Adapt the code to make it "Embree-centric".
* Or else, make the code more modular and flexible, and able to handle several acceleration kernels and not only Embree in the future.

From my perspective, making the code more modular and flexible would be a great advantage for future development. However there are two main problems with it:
* It would duplicate data structures (data structures for the YafaRay objects and transformed data structures for the accelerators). This would mean duplication of memory usage!
* It would need time to convert the data provided to the acceleration kernel (rays, etc) and the data returned from it (rays intersection, interpolation data, etc) into the YafaRay counterparts. This means rendering would not be as fast as it could be.

Making YafaRay Embree-centric (what I have now in my test code) would allow to have the minimal possible memory usage and maximum speed, but at the cost of being inflexible and unable to change or add future acceleration kernels.

The second decision I have to make:
* Make the code easier to understand and maintain? (advantage for developers, but would be less efficient, more memory usage and slower)
* Or else, make the code as efficient as possible? (advantage for users, but more difficult to maintain the code)

The third decision:
* Should we keep our current YafaRay internal "plugins" system? It has some advantages, for example in compilation time, but on the other side I don't really see anybody creating "plugins" for YafaRay. Even PBRT v2 abandoned the original PBRT plugin system. In my opinion, the YafaRay plugins system makes things more difficult (the path to the plugin .so or .dll files has to be known or "guessed" for example). Would it be better to make YafaRay more "monolithic" into just a few .so / .dll files so no need to search for the "plugins" libraries anymore?

And the fourth decision:
* What about the Public API? Right now all the API is in the "core_api" folder but it's probably unnecessary to have so much stuff there? I think the "core_api" should be just the basics needed to do more or less what the Python interface can do. Should it be minimized then? Or even only allow the YafaRay interface to be the public API instead of the current "wide open" one?

Thank you in advance for your feedback!

[David Bluecame]

After thinking about it a lot, I have decided to make the code easier to understand and maintain, and also make it more flexible and not exclusively dependent on Embree.

RAM usage will be worse, as quite a lot of the objects faces/vertices data will be duplicated between the internal YafaRay representation and the representation used by the different possible Acceleration Kernels. Also, the speed might not be as good as it could be.

However, on the bright side, I hope to be able to make the code easier to maintain and extend in the future. I think this is a very important point to be able to keep YafaRay alive in the future.

[Samo]

Your time, your decisions. We users respect that and trust you !!

[David Bluecame]

Thank you for trusting me, Álvaro!

The current v4 code with Embree in my development code is way too messy. It is more a proof of concept than code good enough to be published, and that's why I have not yet uploaded it to the repository.

I'm now in the process of "deconstructing" my code and rebuilding it with better abstraction/encapsulation foundations. Hopefully it will also be more "correct", have a better style and be easier to maintain and more flexible.

What I'm trying to introduce now in the code are abstractions for a new class I've called accelKernel_t. I will initially create a subclass called embreeKernel_t which will contain the interface to convert internal YafaRay objects into Embree scenes/objects, Embree instances and ray intersections.

For now I will not use the acceleration kernels for anything related to interpolation, materials, shaders, etc. Only to do ray intersections and obtain the ray tmax (therefore the hit point), the object ID, primitive ID of the intersection and the "barycentric u,v". YafaRay will process those using "normal" code to get the interpolated textures, materials, vertex interpolation, etc.

This way, in the future new accelKernel_t derived classes could be added with other kernels (perhaps even GPU based) for such intersection calculations.

I hope to be able to reduce any data duplication and transfer to the minimum possible, but if needed I will sacrifice it to favour abstraction. I think it's worth it

[Samo]

The more I see at the landscape of raytracing, the more convinced I am that an agnostic abstraction layer for calling intersection libraries is the best way to go, as more OS acceleration libraries are becoming avaliable. Also this is the only way we could ever consider using GPU cores in a sensible manner (NanoRT, Radeon-rays, etc). Anyway I think this new development will have a deep impact on YafaRay, as future development will focus on rendering rather than on geometry intersection and management.

[David Bluecame]

Hi, Alvaro.

Yes, indeed I was also thinking about RadeonRays for GPU and did some tests creating an intersector for it, but I had problems with RadeonRays itself

RadeonRays uses OpenCL and that is heavily dependant on the OS and the graphic card you have. I could not make OpenCL and RadeonRays work at all, neither in Linux nor Windows with my laptop, so I had to abandon the whole idea. (

But if there are more CPU intersection libraries available in the future, we will be able to integrate them.

[Harvester]

Thank you for your efforts to keep YafaRay current.