Common Evaluation Criteria
Jurors are asked to evaluate your submission using four criteria: Concept, Novelty, Interest, and Quality. The final submission score is based on a combination of these factors. For example, a high-quality talk that has broad appeal and is unlike other recent SIGGRAPH talks has a good chance of acceptance, while a poorly motivated submission of interest to few attendees (or that duplicates recent talks) will probably be rejected.
Concept
How exceptional are the ideas, problems, solutions, aesthetics, etc. presented in this submission? How coherently does the submission convey its overall concept? Is the concept similar to existing ones, or does it stand out? This criterion is particularly applicable to submissions that pull together existing technologies into a single product (for example, demos, animations, art pieces). Submissions of this type, where the individual technologies are not necessarily new but their combination is, are evaluated on both the final product and how well proposed technologies integrate to meet the desired goals. Many submissions in this area are rejected because they do what existing systems do, and they do not demonstrate that the proposed approach leads to better results.
Novelty
How new and fresh is this work? Is it a new, ground-breaking approach to an old problem, or is it an existing approach with a slightly new twist? You must first demonstrate to the jury that your work is sufficiently different from existing approaches. Second, you should evaluate you work in the context of other approaches where appropriate: Is it faster? Easier to use? Does it give better results? Is it more accurate? Many submissions are rejected either because the work is too similar to existing work or because the submission materials did not convince the jury that the improvements were substantial enough.
Interest
Will conference attendees want to see this? Will it inspire them? Are the results or approach appealing to a broad audience? This is partly a measure of how broad the potential audience is and partly a measure of the overall clarity and novelty of the submission. A submission in a very niche area is more likely to be accepted if the results are exceptionally better than what exists already, or if the proposed solution might be applicable to other areas.
Quality, Craft, and Completeness
This is a measure of how well-written the abstract is and the quality of the supporting materials. The abstract must effectively communicate both the problem and the solution in enough detail and clarity that the jury can evaluate it. You must also convince the jury that your solution works. Many submissions are rejected because, while the problem and solution seemed interesting, the materials did not convince the jury that the solution had actually been implemented and evaluated. If your submission has an animation, simulation, or interactive component, then including a video is essential.
During the review process, the jury sorts submissions into three categories: Research, Production, and Studio. Submitters do not need to select one of these categories. This sorting is purely for use in scheduling accepted talks and to provide "at-a-glance" information to attendees. The primary reason for rejection of each type of talk is given below, to help submitters understand what the jury will be looking for in typical talk submissions:
Research Talks
Accepted research talks typically fall into one of two categories: an exploration of a new problem or a novel approach to an existing problem. For talks on new problems, the jury accepts those they believe will interest attendees and inspire subsequent discussions or research. For talks on solving existing problems, the jury accepts those that clearly solve a problem of interest to many attendees. Primary reasons a talk is rejected include:
1. The jury was unconvinced the work solves a new or existing problem. This can happen either because the abstract did not clearly differentiate the work from existing work, or because the proposed solution was too incremental.
2. The jury was unconvinced that there was sufficient improvement over existing work. It is not sufficient for the approach to simply be new; the submission materials must also demonstrate that the proposed approach works better (it is faster, more accurate, uses less memory, easier to use, etc.) than existing work.
3. The submission materials did not clearly convey both the problem and the proposed solution. If the jury has to struggle to understand the submission, they are unlikely to accept it. Good abstracts first provide a concise statement of the problem and solution then provide sufficient detail to convince the jury the submitter would present a compelling talk.
4. The area is off-topic and unlikely to be of interest to SIGGRAPH attendees.
5. The jury believes the talk will be an extended advertisement for a product.
Production Talks
Accepted production talks typically explore solutions to problems frequently encountered in production environments. They should be motivated by unique visual results or the production pipeline rather than production scope, size, or budget. Examples include new applications of research ideas in a production setting, combining existing techniques in new and unique ways, or improvements to pipeline tools or workflow for improved efficiency. The jury accepts production talks that will interest attendees seeking details on production difficulties and their solutions or because the technical details may interest the broader SIGGRAPH community. Production talks are not limited to film and visual effects, but could cover other production environments such as game development, mobile graphics, or CAD software.
The abstract needs to provide context for the work and visual goals, the underlying technical solution, and some kind of evaluation metrics. To support the submission’s claims, we strongly encourage the inclusion of some kind of visual or video material, either work in progress or the finished result. The jury frequently rejects unsubstantiated submissions. Note: it is possible to submit material for viewing only at the jury meeting (when it has not been approved for public display or release). Please contact the Talks Chair to make arrangements.
Primary reasons that a production talk is rejected:
1. The jury was unconvinced the submission provides a substantially new solution to a production problem. Exceptions may be made when the solution is only known to a small community; in this case, clearly acknowledge previous work and explain how this talk reaches a broader audience.
2. Workflow improvements are not supported by an objective measure (for example, a reduction in render time or shot turn-around).
3. Critical visual media are missing, making it difficult for the jury to judge the approach in practice. If media will be viewable only at the jury meeting, make that clear in the submission so jurors understand the images are only stand-ins.
4. The talk fails to cite existing work or explain differences from existing approaches. While talks need not be as rigorous as research papers, a clear discussion of the historical context is important.
5. For large productions with multiple talk submissions, the jury may feel that there is some overlap, and some or all of the submitted talks could be merged into a single, stronger talk. In this case, they will reject one and accept the other, suggesting such a merger. If this is not acceptable, please contact the Talks Chair to discuss the situation.
6. The jury is unclear what they would learn by attending the talk; vague talks are a waste of attendee time. Your submission should provide insight on the talk content.
Studio Talks
The Studio is the place for making and creating at SIGGRAPH 2016. A Studio Talk presents technology or a project that:
- Can be used by attendees from many backgrounds and levels of expertise.
- Facilitates creating and experimenting.
- Is very durable and able to withstand use by attendees for the entire duration of the conference.
- Allows attendees to bring home a digital or physical artifact of their experience.
If your Studio Talk meets the above criteria, it will be evaluated based on the following:
- Benefits to Attendees
How will presenting your technology, process, or project directly benefit attendees? What will they be learning and how can they apply what they learn to their own work?
- Level of technical innovation
- Does the project represent a new step forward in graphics and interactive technology?
