Default weights (Price 25, Quality 25, Lead-time 20, Service 15, Reliability 15) are a starting point. If you sell perishables, bump Reliability up. If you sell commodity industrial parts with multiple substitutes, bump Price up. The tool makes this explicit instead of pretending weights are universal.
Subjective scoring needs a rubric. "Service" without a rubric is whatever the last person who got yelled at scored. Use: response time in hours, RMA turnaround, willingness to do small custom orders. Convert each to a 0–100 with a written rule.
Score once per quarter, not once per crisis. The value of a scorecard is the trend. A supplier whose Quality score fell from 90 to 70 over two quarters is telling you something even if their Price is great.
Limitations. The scorecard doesn't model switching cost. Switching suppliers has a real price (qualification, dual-running, ERP setup). If the leader is well ahead, the gap matters; if they're tied, the gap probably doesn't justify the change.