CFD: a complex mix of Physics, Engineering and Art! Results need validation and sanity checks!
Validation and Verification of CFD Results
The word "Computational" in the phrase "Computational Fluid Dynamics" is simply an adjective to "Fluid Dynamics". Hence, while dealing with aspects of CFD tool or process, it is vitally important to keep the physical understanding of fluid dynamics uppermost in user's mind as CFD has to do with physical problems.  Adapted from John D. Anderson, Jr (Computational Fluid Dynamics  The Basics with Applications).
Any numerical simulation process is not just "Meshing, setting Boundary Conditions, Running Solver and Making Colourful Contour / Vector Plots". The results ultimately needs to be converted into a set of inputs for a robust design of the component of system. The sound knowledge of "underlying fluid mechanics principles and operating conditions of the problem setup" are more important than just knowing how to use the software. Some of the requirements which will help a "CFD practitioners" take correct design decisions based on CFD results are:
 Basic of Fluid Mechanics such as Bernoulli's Equation and actual physics behind this principle
 Fluid Mass Conservation and Momentum Conservation with underlying mathematics
 Shear stress formulation on fluid flow, Empirical Calculations, Experimental Data
 Flow behaviour such as Developing Flow, Developed Flow, No Slip Condition, Separation and Reattachment
 Complete theoretical and experimental behaviour of following flow conditions
 Flow inside a circular duct
 Flow over a circular cylinder
 Flow over a flat plate
 Flow between two parallel plates
 Natural Convection in an Enclosed Cavity
Though the industrial flow configurations are far from being closer to these simple geometries, the fundamental ideas contained in them are indispensable to a good understanding of modern computation methods. The methods and results arrived at are important not only for these simple flow but also for the extension of our fundamental knowledge of turbulent flows in general. Methods for dealing with turbulent flow of
any industrial applications could be devised only on the basis of the detailed experimental results obtained for them.
For example, according to measurements performed by H. Kirsten, the entrance length of a turbulent flow in a pipe is about 50 to 100 diameters. This knowledge is very important in deciding the inlet boundary condition for any industrial internal flow configuration.
CFD: Computational or "Colourful Fluid Dynamics"?
CFD is a great tool when used with appropriate procedure and guidelines because of its inherent nature of multidisciplinary science leading to technically unlimited potential and applications. Yet, "CFD is not a panacea of all your Flow and Heat Transfer problems without experiencebase insight". Any result must be looked at by an experienced engineer in that field and must go through an "orderofmagnitudecheck" before accepting the results.

CFD simulations are capable of predicting good qualitative results (trends). It will not make decisions for design engineers but certainly help them take more informed judgment. When no information is available about flow structure in a system, CFD is certainly an economical start into detail analysis of the performance of the system.

Even inaccurate CFD results, so long it is ensured mathematically physical, possesses many features which make it very useful:
 The sheer capability of detailed visualization is rich in information.
 CFD results give an insight which is not possible by experiments and other theoretical means.
 Trends are usually reliable and leads to right direction in terms of design evaluations.

In many cases, quantitative information is predicted with sufficient accuracy to justify
engineering design changes on commercial plant. CFD can even predict more useful information
than testing because the measurement point (typically based on user experience) may not be at
appropriate location.

Historical knowledge obtained from plant operation is a great validation tool for such numerical
(also called Virtual) simulation results.

Notwithstanding the limitations mentioned above, CFD models drastically reduce implementation of "Design Modifications & Scaleup of a System"
 CFD can be also used early in the design stage for performance evaluation, for optimization and enhancement during the development stage and for diagnostics in the later stage of the produce development cycle.
Excerts from H. Schilchting
All methods for the calculation of turbulent boundary layers are approximate ones and are based on the integral forms of the momentum and energy equations. Since, however, no general expressions for shear and dissipation in turbulent flow can be deduced by purely theoretical considerations, it is necessary to make additional suitable assumptions. These can only be obtained from the results of systematic measurements and, consequently, the calculation of turbulent boundary layers is semiempirical.
While the usage of CFD simulations in industry is on rise at rapid pace, the credibility of results of any such calculation is still an area of concern. Most organizations using such codes, over time have evolved their own best practice guidelines to minimize the chances of "critical errors". There are many such guidelines issued by ERCOFTAC and AIAA. Following diagram summarizes classifications used to designate the types of error which needs to be addressed when CFD simulations are used to make design decisions beyond extant knowhow of the company.
Steps to check and increase accuracy of CFD results
 Step01: use of monitor points  use monitor points relevant to the simulation physics to check if simulation is proceeding "as expected". The term "as expected" has been to emphasize the fact that one should always have some idea what he is expecting from a CFD simulation!
 Step02: compatibility of wall function and areaaveraged y^{+} value  each type of wall function developed has certain requirement on boundary layer resolutions. Ensure that the areaaveraged values on all the walls fall in that range  areaaveraged because it is not always possible to maintain the y^{+} in a certain range for every location of the walls.
 Step03: sensitivity to mesh size  you have chosen a particular mesh size on walls and in the fluid region based on some reference or past experience or even due to constraints on available hardware (machine RAM size and CPU speed). It is advisable to first check the effect of element sizes on a mesh coarser by two times the baseline mesh.
 Step04: effect of turbulence values at inlet  most of the time the turbulence value specified at inlet face is a guess, say tubulence viscosity ratio = 5 and turbulent intensity = 10. Do we know where these values have come from? Check if the results are affected with these values are doubled or halved. Ideal situation would be to have the turbulence level based on measurements but it is not always the case.
 Step05: effect of temperature dependent material properties  this is important espcially when the temperature gradients are 'large' or the change in pressure is such that it can lead to 'significant' change in density of gas. The value of 'large' and 'significant' used here changes across the applications, a value of 10% is considered to be 'significant'.
 Step06: extraction of quantitative values and interpretation of results  sometimes the way areaaveraged or surface integrals are calculated, they need to be interpreted appropriately. For example, even in case of a constant area duct with water as incompressible constant property fluid, the change in static pressure between inlet and outlet will not be same as change in total pressure.
 Why? As the area is same and fluid is incompressible, the expected changed in dynamic head (velocity pressure) is zero!
 The explaination lies in the way areaaveraged or massaveraged dynamic head is calculated at outlet where flow is not uniform whereas the specified velocity field at inlet is uniform.