View topic - yafaray issues and suggestions

yafaray issues and suggestions

General questions related with the YafaRay Project, 3D computer graphics and about this site.

yafaray issues and suggestions

Post Sat Mar 27, 2010 3:12 pm

yafaray issues and suggestions
or
What breaks the yafaray workflow?

Due to some discussion in IRC I put together my thoughts on usability of yafaray. I was asked to offer sane professional points and esspecially not a hobbyist approach...

Please judge for yourself and add your critics to the beforementioned (on how I performed on that) as well as the points I outline as follows. Some of my critics apply to organization, others to responsiveness and professionalism. Before you answer: "Its my free time", "This is a free poject",... Please keep in mind that all of you got a goal with yafaray. For the common goal as a team some structures might be needed. This might also include a more professional approach to some areas which includes the appropriate tools.

Summary

In its current state yafaray doesnt offer a smooth workflow, which is directly related to lack of usability. This post lists critics and issues categorized in top level topics from general issues, over workflow issues, specific technical and render issues up to marketing and overal appearance. In a next step those issues are put together into individual problems as of today with yafaray.

That way I point out the reason on why something is going wrong - not the symptom. Additional questions offer ample opportunity to understand the listed problem.

Final results are listed in Chapter III. - all 3 points are put together in a way that by completion of all points all issues would be solved:
1. as commandments aka development/application/design guidelines
2. as organizational requirements to enable above commandments
3. as requirements for next yafaray features

To save time you can directly jump to Chapter at the bottom. Upon any question to a point in Chapter III you can read the additional info in Chapter I - II.

Content:
I. Critics and Issues
II. Consequence and underlying Problem
III. Suggestions - heretic's yafaray commandments

III. is my personal opinion, yet it is meant to be a basis for a design and coding/usability guideline book. Something which should be (in a modified final form and agreed upon by all involved people) carved in stone for yafaray changes.

Critics
A Workflow
A1 yafaray/blender workflow for still images is not smooth, but interrupted by a lot of issues. This is mainly due to all those small single issues from point B onward. All together force to stop a render by times, rerender, think of new tricks to achieve the desired result,...
A2 yafaray/blender workflow for animations (image sequences) is not smooth, but interrupted by a lot of issues. Some - even important - things are not possible in blender/yafaray.
A3 yafaray support for postpro is slacking in some departments hence some postpro tasks are more difficult than with other environments.
A4 Render workflow suffers from interruptions - see D2
A5 BG, IBL issues stop a smooth workflow - see E1
A6 Volumetrics see F3

B. General Issues
B1 It seems yafaray dev team is more focused on features (to come up par with other renders or to offer the most capable render) than on stability and smooth workflow.
B2 Bugtracker - to any user without dev contact the bugtracker is meaningless. None knows if anyone saw the issue and if somethign might happen with it or not. This leads to B3.
B3 Lack of reliability for users without dev contact. It is impossible to count on something happening.
B4 Integration with the 3D program - the current integration with blender has specific weak points, which in return result in more issues (some of which are listed here).

C. Materials
C1 Material consistency is weak. Some features, which are available in one material, are not available in another material.
C2 yafaray blend material/node system behaves unexpected and unreliable.
C3 yafaray blend material appears to be inconsistent.
C4 yafaray does not support all procedural noise texture features of its current (only) host application (blender). Some of the missing features can safely be assumed to be important.
C5 glass material has issues.
C6 frosted glass (current DT dev branch) has issues. This issue is partially related to lack of SSS.
C7 yafaray doesnt offer "real" SSS. This is imho a major showstopper for quite some new users.
C8 A personal note: to me it "feels" that the emit property has issues in lightning a scene.
C9 Lack of Halos and related materials.

D Render
D1 yafaray has unexpected differences between lightning methods.
D2 yafaray render workflow is more "try and error" than exact science. Unexpected behaviour of render results adds to this. Something which in return results in a showstopper for smooth workflows. (examples: adding volumetric can change the whole render setup for a scene, where at first 10-20 pass render with 1-5 samples was feasible, the whole render strategy changes to strong noise reduction in first pass, which might also change sample rates of light sources,...)
D3 Bidirectional Tracing appears to be the stepkin of yafaray
D4 Render times increase due to lack of instancing
D5 High render memory usage esspecially due to "makign real" of particles and/or lack of isntancing.
D6 Its close to impossible with current average machines to render 8MP (MP=million pixels) images, 16MP images can be assumed as close to impossible and 32MP is a nogo due to memory usage. 8 MP, preferably 16MP images are industry standard to create images for sale or for print. 32MP is soon a must-have.
(Dx Some people would post the lack of irradiance caching here. I disagree on that point since irradiance cache only helps with some or most still images, but is imho unusable for animations due to irradiance flickering between frames. Therefore it is a nice-to-have feature but nothing for a yafaray critics document.)

E World and IBL
E1 environment maps/background and IBL texture (HDRI) lightning are treated as different topics in yafaray, partially as postpro only issues. This results in severe issues for a general production workflow.
E2 volumetric implementation is "clumsy" and lacks in some departments.
E3 Handling of sunsky and DT sunsky is clumsy to deliver expected results.

F Objects
F1 Meshlights have issues.
F2 underlying material of meshlights has issues eg shadows on a meshlight material.
F3 Volumetric is close to beign unusable in a production workflow due to render time and lack of exact control from within the 3d host application - latest when using noise based volumetric for an animation (besides some scatterign and absorption in preferably square rooms as well as exp density volumetrics in some situations).

G Documentation
G1 Beginner documentation for new users is non-existant (btw: this is my fault since I promised to write one and already got the first two chapters done).
G2 Crucial information is spread over several documents and tutorials.
G3 yafaray lacks guided tutorials, where a user is instructed to do something including proper explanation to achieve an expected result on his own (quick wins for a new user, like my first render, my first scene, my first yafaray Gignerbread man,...).
G4 Some documentation remarks are inconsistent with current yafaray version.
G5 Overall the yafaray documentation leaves the feeling of "patchwork".

H Marketing
H1 yafaray logo has issues. The graphic as such lacks proportion (too much width). The name (yafaray) is too small compared to the logo. From a marketing point of view this needs to be changed (example: Blender and yafaray logo of same size on a render result bottom corner - by then a viewer can read blender, but not yafaray text in yafaray logo).
H2 Marketing appears to be a single man effort. Basically it looks as if Samo does "things", pushes them forward and nothign would happen if he wouldnt. From the outside it doesnt look like a group effort. If this it completly or partially true, then this is prone to failure - latest when Samo gives up on his own (or god-forgive somethign happens in rl).

I Development
I1 In regards to development I only got impressions and no real basis to judge since I lack insight. Still I dare tell that yafaray development "feels" like a bunch of "senior wizards: render engine". Unfortunately this implies that each wizard has his own tower and his own experimental environment - working alone for the most part. It esspecially doesnt offer the feeling of a well-coordinated group of professonals working together as a team to provide a team effort. (Mark me its an impression I got and I value and trust each of those devs I learned to know, but... for now I wouldnt trust you as a team - since it doesnt "feel" like a team.)


II Consequence and underlying Problem
A Workflow
Consequence: The yafaray workflow is steadily interrupted and seldomly smooth.

Underlying Problem 1: The yafaray design doesnt include "failure of yafaray and/or blender". It is imho crucial to design yafaray as if each single library, procedure, function,... is buggy and forces blender and/or yafaray to crash resp. to corrupt data. Please keep in mind: yafaray becomes increasingly complex and some settings are hours of work by now, maybe even days. Consider a designer who works two days remodelling and who saved corrupt material data over those two days. Aka no backup - first render after two days would lead to a crash with yafaray data lost.

Questions:
- Where is the code to preserve work done so far by the user until a bug or crash happened?
- yafaray-only-autosave?
- repair a 3d host file automatically?
- trace faulty materials (or more general: settings) and delete those, maybe saving existing settings of that material to a txt-file? (auto-repair)
- import consistency/version check of yafaray settings (eg when importing or linking materials with yafaray attributes)
- (remark: did anyone consider writing a 0.1.1 to 0.1.2 conversion script?)

Underlying Problem 1a: Closely related to 1 - yafaray doesnt offer to (auto-)save during rendering.

Questions:
- Why doesnt yafaray autosave each pass?
- Why cant I add more passes and continue with a saved render? Eg all passes done - render finished but I realize 5 more passes are needed, maybe with lower pixelwidth even.
- Why is yafaray so memory-hungry?
- Why does yafaray crash so easily upon yafaray "memory-hungry" computations during render to QTGUI?
- ...

Underlying Problem 2: yafaray integration with blender is clumsy and yafaray offers features, which blender doesnt have (and vice versa).

Questions:
- Why cant I set glass absorption and glass filter color by using eg. color channel (and another eg. spec for the 2nd)?
- More general: Why cant I set EVERY yafaray slider, option,... from within the hosting 3d app? (This is a major reason for a lot of issues - it also is more of an issue in Bledner 2.5/2.6 where "everything" can be animated).

Underlying Problem 3: yafaray developer "assume" what users need and set an options dimension accordingly.

Questions:
- Why is glass dispersion NOT a 4 digit value? values as low as 0.001 and maybe even lower make sense.
- Why cant I set volumetric step size lower than 0.1? This is needed in small scenes for smooth renders.
- Why is Brightness in DT sunsky NOT a 4-digit value? Interesting for dimly lit scenes.
- Why is Emit 0.1 the lowest possible value?
- Why is IOR 1.00 not a 4-digit number?


B General Issues

Underlying Problem 4: Visibility of the dev team to users. This includes responsiveness and reliability.

Questions:
- Why arent crashes always set to critical immediatly?
- Why are issues in Bugtracker without response for weeks?
- Why are there unassigned issues on page 1 in Bugtracker for weeks?
- Why does none feel responsible to assign a fix date for critical issues? (I know some do, but none outside, who doesnt know a dev, has that impression..)


Underlying Problem 5: Integration with hosting 3d application is slacking.

Questions:
- Why doesnt yafaray always use the 3d applications settings as default values? Eg. diffuse color,...
- Why doesnt yafaray update the hosting 3d application? (eg difuse color to identify objects more easily in 3D view)
- Why arent additional channels available for each material? Eg. Transparency for every material; same for translucency,...
- why dont yafaray objects derive starting settigns from blender at reasonable conversion multipliers? (an area light in blender can only be an area light in yafaray - still I need to choose the option in the exporter :? , why cant I set lamp value(s) in blender - energy, color,... its - the same :( )
- Why do I need to read through all yafaray documentation only to realize after 2 days, that eg. yafaray doesnt use the default Blender noise? ... (I had this after reading yafaray documentation twice in my first week - you simply dont know the whole documentation by heart after readign once or twice - esspecially basic blender noise is expected to work by everyone who is new to yafaray and who didnt by chance realize that yafaray doesnt support it - add lots more examples here)
- ...

C Material

<Insert Underlying Problem 5 here too with following remark:> Materials are inconsistent.

Questions:
- Why is there no transparency (alpha channel) for glossy?
- Why is there no color channel for glass? (Right now one MUST model stained glass in 3D, whereas using a texture would be easiest)
- Why are rough glass shades dark? One would expect both possibilites aka dark shaded glass and "milk glass"
- We live in the Internet information age! Why is there no material repository? Why cant I access the online material repository from within yafaray exporter? Why doesnt yafaray offer me a smart solution to my own or a community material library from within the application?
- Why are settings from blender material so much different for yafaray - even for almost identical materials? (eg simple shinydiffuse red with a bump and a spec map)
- ...

Underlying Problem 6: The Nodes system behaves unexpectedly. (Most of this might be due to current Dev branch - so take this one with a huge load of salt)

Questions:

- Why does the blend texture darken the blend material?
- Why does the blend material at 0.0 show the exact material 1 but at blend 1.0 not material 2 instead a black 100% absorption material?
- Why cant I mix differing IOR glass materials? (not only my experiments, but consider also boiling water, hot air over a street, astronomic refraction,...)
- remark: While at it - why isnt there astronomic refraction in DT sunsky?
- ...

Underlying Problem 7: yafaray lacks some effect materials. Some effect materials behave "weird". It seems as if some effects are inconsistent to other features.

Questions:
- Why arent there Halos?
- Why cant I render lens flares? (yes you can postpro in a still - now make the same for an animation?)
- Why doesnt emit light a scene like eg a lamp? (at first there is a huge difference between those. Take this example: glowing red hot steel is a candidate for emit, even hotter steel at almost white color is a candidate for a meshlight. Where does one end and the other start?)
- Why has emit light power 20.0 less effect than a lamp with 1.0 power? Even though the emit object is larger than a lamp?
- Why are there shadows on mesh lights? (on emit shadows are ofc ok to some degree, but a bright meshlight imho shouldnt have shadows on itself.)

Underlying Problem 7a: glass material has issues.

Questions
- why isnt the glass IOR mapped to a texture channel (related to the general issue to attach all yafaray sliders to the host 3d app)? (loads of examples of its use above and below)
- glass offers absorption - why doesnt glass offer scatter? (eg. milk glass - NOT SSS)
- lets assume glass would offer scatter - why dont you add texture channel for volume pattern (while at it one could even use this for the alpha channel issue, hence offerign a really complex alpha channel playground for all materials)? (eg astronomic refraction - using glass for atmosphere, simple glass structures inside glass, glass engravings, raw emeralds, raw quartz, crystalline structures, boiling water,... - all this must be modelled as it stands now, but such a feature would be grand)
- insert questions from above problems on rough glass, IOR blend,...
- insert questions on lack of alpha channel, lack of assigned color channel,...

D Render

Underlying Problem 8: Inconsistency of Lightning methods.

Questions:
- Why is a scene in DL default setting dimly lit compared to the same scene in PM default setting? (yes am aware, its totally different - consider this: one switches Lightnign method and must readjust eg 20 light sources power individually, even though the relation of all light's power to each other was perfect... - a simple global light power slider would help, which changes every light source power percentage-wise or linear.)
- Why are PM default settings so much different to PT (with Photon Map) default settings? Why are the photon map fields in PT different to PM? (I only refer to the similar values/fields)
- Why are Output settings part of the Settings window? Or to be more explicit: why are Lightning method settings not seperated from Output settings? (The settings pane is quite crowded as it stands now..)
- ...

Underlying Problem 9: Render of the underlying mesh(es)/scene has huge requirements - more than eg blender internal.

Questions:
- Why does yafaray not offer instancing? (sorry, real object is real object - what for the high mem usage for a lot of the same meshes? VRay solved it with proxies.)
- Why does yafaray make particles "real"? (closely related to instancing - sure making a 1d or 2D particle a mesh or at least extruded curve is cool, but making a house in a meadow scene jump from 100.000 verts to 5 Mill verts due to "makign grass real" is unreasonable - its just not smart).
- Why can I render a 16MP image with blender internal (even fast) but yafaray crashes already during creation of photon map?
- How do I make a 32MP image with yafaray on Win32 with 2-4 GB RAM?
- ...

E World and IBL

Underlying Problem 10: World settings are not well thought out.

Questions
- How do I align HDRI, background image and IBL diffuse light? (yeah this refers to sIBL later on - sorry but I consider background in postpro not the "always 100% solution" - this hurts esspecially in animations!)
- Why does yafaray lack a preview for sunsky and DT sunsky? (there was a website with an applet to show sunsky options of yafaray in each setting...)
- Why arent there presets in sunky and DT sunsky? (eg. time/environment/atmosphere/altitude, example three preset tabs offerign different settings: night, sunset, morning, noon,../foggy,../clear, dusty, sandstorm/ground, ocean-level, 1,000 ft, 3,000 ft, ...)
- Why does IBL sample compete by default with light source samples? ( a general power/sample slider would make sense which adjusts all light sources - even emit - but which can also be turned off...)
- ...

F Objects (all others in F have been handled already)

Underlying Problem 11: volumetric is clumsy and lacks features.

- Why cant I make "simple" volumetric effects with volumetrics? (faint thing heat from a cup of coffee raising, jet plane trails as volumetrics,...)
- Why is volumetric bound to a cube?
- Why is volumetric so computing intense? (Yes am aware its not easy, but... right now it appears to be "too much" when one needs a clear volumetric render)
(remark: my personal question remains: I posted some papers for near rt volumetric cloud calculation including reference papers with formulas - DarkTide worked with them for some time afaik, apparently with a good test run even - whats happening now?)

G documentation

Underlying Problem 12: yafaray is complex (or appears to be) and lacks a consistent documentation directive.

Question:
- Why isnt there a hint in the host app, that I cant use that feature in yafaray?
- Why doesnt yafaray offer context help except a few comments on mouse over? (eg. a help button for each pane in exporter to jump directly to the correspondign yafaray doc would help already new users.)
- Why isnt there a yafaray wiki?
- Why isnt there a noob to pro guide? (yes my fault - this truly is!)
- Why is yafaray "complex" coupled with the "patchwork documentation"? (It feels complex because one needs to learn yafaray documetnation by heart as a beginner or learn/live by mistakes the hard way.)

H Marketing

Remark: "Marketing" might appear to be out of place here. Yet it is meant to attract new users from outside. This would also solve some tutorial and documentation issues. More users = more people willing to make a vid tutorial or write more guides,...

Underlying Problem 13: yafaray lacks a visible vision/goal and the approbriate marketing concept.

Question:
- Who looks after consistency and proper logo usage?
- Who makes sure that design and project fit to each other?
- Who makes sure that yafaray makes current users feel more comfortable (mostly architectural professional-wise atm)?
- Who knows current yafaray average user profile? (Is it really architects only for professional work? Mostly hobbyists? Mostly Pro's in their spare time,...? Knowing this, one could decide, which feature is more important to current user base or to a future target user base...)
- Who defines the next target user group? (Maya users? blender animators? All architects? Professional graphic designers who casually work with 3d? Professional 3D designers who need a free render? ...)
- Why is condars "yafaray vid" not linked on the frontpage? Where is Aspen's demo video - it is not on frotnpage either?
- Where are the efforts to convince blender foundation to encourage more yafaray users? I dont read a word about yafaray by eg. Durian staff.
- Is there any company in existance who uses yafaray? Which? Why dont we have a "reference list"?
- Last architect challenge no yafaray user particiapted. Where is the Marketing guy who persuaded the organisator to add yafaray out of competition and where is the Marketing guy who started a yafaray community thread to make a yafaray render an out of competition community effort (while still making sure it is shown in the competition)? A competition offers a broader awareness - we all know that. Yet some probably thought: there were test renders on yafaray site so those will make it (I thought so).
- ...


III. Suggestions - heretic's yafaray commandments

From above points my suggestion of commandments, which imho solve core problems if obeyed and followed 100% in descending order of priority:

Application/Development commandments
===========================


  1. yafaray is software. Software is always buggy. User data is the most precious thing a software has. We do everything humanly possible to preserve and save user data anytime even if yafaray crashes.
  2. We check yafaray data consistency in the host 3d application and warn the user of issues before they become a problem.
  3. yafaray must be easy to use. We avoid complex windows/tabs with too many settings (current settigns panel in exporter is my prime example together with World panel esp. with DT sunsky settings)
  4. yafaray offers automated/semi-automated error correction based on the previous point. (ultima ratio: save all yafaray data to a human readable text file and delete all yafaray data in the host 3d application scene afterwards. This is to be refined over time as we learn about additional issues.)
  5. Fixing yafaray crashes and user data loss is always of high priority for all devs.
  6. Coding goes with documentation. Latest after accepting a feature to the official branch and after successfull testing, a documentation update is due for this feature. For this we maintain two documentations: 1. the last stable documentation, 2. the official DT-branch documentation. Documentations/updates can be written by anyone, preferably a beta-tester approved by the developer in charge of the respective feature.
  7. We value any user input. As such all devs make sure all entries in Bugtrack are handled within 24 hours. "handle" explicitly includes answers like: "More info please", "Post example file", "Post your system config",.. For this we define the handling of bugtrack entries enabling us to close entries asap.
  8. We offer and maintain a development roadmap, which is public. Approximate deadlines are included.
  9. Yafaray materials always offer all possible texture channels. Eg. all yafaray materials have an alpha channel, all materials use the color channel, all materials use the mirror channel,...
  10. All yafaray materials, objects and settings MUST be logically consistent to each other. (all texture channels possible, switchign lightning methods produces similar results in regards to lightning maybe by means of an overall light source power slider, similar layout and default settings of PM and PT/pm render setting, all sliders are either full numbers OR offer exactly 4 digits after the period, light is light - we make sure light power/values offer consistent power at same numbers over all light sources,...).
  11. All yafaray material settings are mapped to the host 3d application.
  12. yafaray always makes ALL features, sliders and options available to the user of the host application. This *may* exclude only render and output settings (some of those might make sense to be available still).
  13. Where possible yafaray directly derives settings from host application settings and updates host application settings vice versa. This auto-update can be switched off or set unidirectly in any direction selectively for each single button/slider/...
  14. yafaray informs the user inside of the host application which settings are available to yafaray and which not (eg. change color of sliders/buttons which yafaray "likes")
  15. We aim to offer the best material repository which ever existed in 3D - even independant from the host 3d application. In future we want a public material repository on our website, which is directly accessible from the yafaray exporter in the 3d hosting application for everyone. This also includes people/companies to add additional repositories based on our technology(eg. their own repository inside yafaray exporter, while still having also access to the official repository).
  16. Alternatively to the last and additional previous points a database approach might make even more sense (either linked to Blender or preferably a standalone yafaray db). Any Client with yafaray installed could connect locally and/or remotely to any number of dbs, where any material, setting (eg. World settings),... are stored as well as any derived (children) settings from those. Considering a low complexity db design (maybe even one table design + remote table cached locally) this could be done without any specifc db (to not add an additional db code base). Imagine... imagine any material, any world setting any volumetric setting,... from any repository (from private repository ones only if the user sets it shared) being available anytime seamless, so that it looks to any user like a single repository... saving yafaray data in the host 3D application file would only be an "emergency fallback" and to provide consistency between file and db. (there are loads of advantages to do it this way - its seamless, you could add rules to encourage users to share all their materials, you would offer the first 100% scalable render for any size of project, you would offer users a smart way to build their own library without thinking about one,...)
    yes this is a full fledged project - I know

Organizational commandments
====================

A draft suggestion for tasks and positions (I know most exist - I outline those here for consistency).

- yafaray board elects one person as "head of yafaray", short "hoy". The HOY :D ensures that commandments are followed by all involved people. Whenever neccessary the HOY demands review and update of the commandments. Basically HOY has the fun in itnerviews but is also the maiden to make sure all tigns are runnign smooth (fame and fortune got its price).
- head of dev, short "HOD"
- head of marketing (H Marketing is yours)
- head of beta-testing/documemtation/QS (Make sure those devs have fun coding and follow deadlines, give you and your people/testers the info you need, maintain documentation, review and test documentation by times,...)

Future feature recommendation
====================

Coupled with the commandments these features will solve all issues I outlined above. As such I recommend to implement these features in descendign priority. The categories of real project and minor projects is my personal view based on my tidbits of info about yafaray development:

"real projects"
---------------

  1. instancing
  2. progressive PM
  3. SSS
  4. implement sIBL
  5. add effects esspecially halos (including lens flares,...)
  6. smart rendering/find solution for 16MP+ renders/reduce mem usage for rendering/enable a more sophisticated tiled render (eg. render a 16MP image in 256 single tiles which are put together afterwards)

minor projects
----------------

  1. overall light power slider (to enable a smooth transition between Lightning methods withotu changing all light sources)
  2. improve world settings
  3. smarter World settings with defaults and presets
  4. improve glass material (scatter, glass volumetric, rough glass shader alternatives,...)
  5. improved volumetrics

------------------------------

This whole document is put together with my current knowledge of yafaray. Some points might irritate you or appear to be wrong (some actually probably are) - some of those might need more explanation, so please ask.

Yours

Martin
Kunden-besser-informieren.de

Images posted by me are solely posted for the purpose of the yafaray project. Any other commercial or non-commercial use, modification etc. of my images and its content requires my written permission.
heretic
 
Posts: 255
Joined: Wed Nov 25, 2009 1:35 am

Re: yafaray issues and suggestions

Post Tue Apr 13, 2010 10:31 am

:shock: :shock: :shock: :shock: :shock:

Big analisys, great work! :)
Yafaray needs this kind of realistic aproach.
Great work once again!

This document should be a "must see" for proposals at gsoc 2010 :wink:
Manuel_Ramos
 
Posts: 166
Joined: Fri Sep 19, 2008 10:14 am


Return to News & Discussion



Who is online

Users browsing this forum: No registered users and 2 guests

cron